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Abstract 



Results are presented from a search for a fourth generation of quarks produced singly 
or in pairs in a data set corresponding to an integrated luminosity of 5 fb _1 recorded 
by the CMS experiment at the LHC in 2011. A novel strategy has been developed 
for a combined search for quarks of the up- and down-type in decay channels with 
at least one isolated muon or electron. Limits on the mass of the fourth-generation 
quarks and the relevant CKM matrix elements are derived in the context of a simple 
extension of the standard model with a sequential fourth generation of fermions. The 
existence of mass-degenerate fourth-generation quarks with masses below 685 GeV is 
excluded at 95% confidence level for minimal off-diagonal mixing between the third- 
and the fourth-generation quarks. With a mass difference of 25 GeV between the 
quark masses, the obtained limit on the masses of the fourth-generation quarks shifts 
by about ±20 GeV. This result significantly reduces the allowed parameter space for 
a fourth generation of fermions. 
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1 Introduction 



The existence of three generations of fermions has been firmly established experimentally [JJ. 
The possibility of a fourth generation of fermions has not been excluded, although it is strongly 
constrained by precision measurements of electroweak observables. These observables are 
mainly influenced by the mass differences between the fourth-generation leptons or quarks. 
In particular, scenarios with a mass difference between the fourth-generation quarks smaller 
than the mass of the W boson are preferred and even fourth-generation quarks with degener- 
ate masses are allowed B2H3J. 

A new generation of fermions requires not only the existence of two additional quarks and two 
additional leptons, but also an extension of the Cabibbo-Kobayashi-Maskawa (CKM) [4, 5 J and 
Pontecorvo-Maki-Nakagawa-Sakata (PMNS) |6l |7| matrices. New CKM (quark mixing) and 
PMNS (lepton mixing) matrix elements are constrained by the requirement of consistency with 
electroweak precision measurements 03 . 

Previous searches at hadron colliders have considered either pair production or single produc- 
tion of one of the fourth-generation quarks I19 HT5I . The most stringent limits exclude the exis- 
tence of a down-type (up-type) fourth-generation quark with a mass below 611 (570) GeV ffl4l 
[TBI . These limits on the quark mass values enter a region where the coupling of fourth-generation 
quarks to the Higgs field becomes large and perturbative calculations for the weak interaction 
start to fail, assuming the absence of other phenomena beyond the standard model |16|. To 
increase the sensitivity and to use a consistent approach while searching for a new generation 
of quarks, we have developed a simultaneous search for the up-type and down-type fourth- 
generation quarks, based on both the electroweak and strong production mechanisms. 

If a fourth generation of quarks exists, their production cross sections and decay branching 
fractions will be governed by an extended 4x4 CKM matrix, V^^, in which we denote the up- 
and down-type fourth-generation quarks as t' and b' respectively. For simplicity, we assume a 
model with one free parameter, A, where < A < 1: 
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The complex phases are not shown for clarity. Within this model, mixing is allowed only be- 
tween the third and the fourth generations. This is a reasonable assumption since the mixing 
between the third and the first two generations is observed to be small |T7). 

With this search, we set limits on the masses of the fourth-generation quarks as a function of 
A. Since \J A = |Vtb|> the lower limit of |V t b| > 0.81 from the single-top production cross 
section measurements [181 translates into a lower limit on the mixing between the third- and 
fourth-generation quarks in our model of A > 0.66. 

Using the data collected from y/s = 7 TeV proton-proton collisions at the Large Hadron Collider 
(LHC), we search for fourth-generation quarks that are produced in pairs, namely b'b' and t'V, 
or through electroweak production, in particular tb', t'b, and t'b', where the charges are omitted 
in the notation. While the cross sections of the pair production processes do not depend on the 
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3 Event selection and simulation 



value of A, the production cross sections of the tb' and t'b processes depend linearly on (1 — A), 
and the single-top and t'b' cross sections on A. 

We assume the t' and b' masses to be degenerate within 25 GeV. In the case they are degenerate, 
they will decay in 100% of the cases to the third-generation quarks, since the decay of one 
fourth-generation quark to the other is kinematically not allowed. However, even for non- 
zero mass differences, the branching fractions of the t' — > bW and the b' — > tW — > (bW)W 
decays are close to 100%, provided that the mass difference is small |T9| . For instance, for a 
mass splitting of 25 GeV, and for V t /y = 0.005 (which would correspond to A = 0.99975 in our 
model), less than 5% of the decays will be b' — > t'W* (in the case m t t < m h i) or t' —> b'W* 
(in the case m t i > my). For larger values of V^, the branching fractions of b' — > t'W* (or 
t' — > b'W*) decrease even further. Therefore, the decay chains remain unchanged as long as the 
mass splitting is relatively small. We expect the following final states: 

• t'b -> bWb; 

• t'F -> bWbW; 

• b't -> tWbW -> bWWbW; 

• b't' -> tWbW -> bWWbW; 

• b'b' -> tWtW -> bWWbWW. 

These decay chains imply that two jets from b quarks and one to four W bosons are expected in 
the final state for fourth-generation quarks produced both singly and in pairs. The W bosons 
decay to either hadronic or leptonic final states. Events with either one isolated lepton (muon 
or electron) or two same-sign dileptons or three leptons are selected. The different production 
processes are classified according to the number of observed W bosons. 

2 The Compact Muon Solenoid detector 

The central feature of the Compact Muon Solenoid (CMS) detector is a superconducting solenoid, 
13 m in length and 6 m in internal diameter, providing an axial magnetic field of 3.8 T. The 
inside of the solenoid is equipped with various particle detection systems. Charged particle 
trajectories are measured by a silicon pixel and strip tracker, covering < cp < 2rc in azimuth 
and < 2.5, where the pseudorapidity rj is defined as — ln[tan(0/2)], and 9 is the polar angle 
of the trajectory with respect to the anticlockwise-beam direction. A crystal electromagnetic 
calorimeter (ECAL) and a brass/ scintillator hadron calorimeter surround the tracking volume 
and provide high-resolution energy and direction measurements of electrons, photons, and 
hadronic jets. Muons are measured in gas-ionization detectors embedded in the steel return 
yoke outside the solenoid. The CMS detector also has extensive forward calorimetry covering 
up to | ?/ 1 < 5. The detector is nearly hermetic, allowing for energy balance measurements in 
the plane transverse to the beam directions. A two-tier trigger system selects the most interest- 
ing proton collision events for use in physics analysis. A more detailed description of the CMS 
detector can be found elsewhere [20 1. 

3 Event selection and simulation 

The search for the fourth-generation quarks is performed using the a/s = 7 TeV proton-proton 
collisions recorded by the CMS experiment at the LHC. We have analyzed the full dataset col- 
lected in 2011 corresponding to an integrated luminosity of (5.0 ± 0.1) fb _1 . Events are selected 
with a trigger requiring an isolated muon or electron, where the latter is accompanied by at 



3 



least one jet identified as a b jet. The muon system, the calorimetry and the tracker are used 
for the particle-flow (PF) event reconstruction (21]. Jets are reconstructed using the anti-fcx 
algorithm [22j with a size parameter of 0.5. Events are further selected with at least one high- 
quality isolated muon or electron with a transverse momentum (px) exceeding 40GeV in the 
acceptance range \t]\ < 2.1 for muons and \r/\ < 2.5 for electrons. The relative isolation, I re \, is 
calculated from the other PF particles within a cone of AR = ^(A<£) 2 + [At]) 2 < 0.4 around 

the axis of the lepton. It is defined as 7 rel = (E T harged + E? hoton + E^ eutral )/p T , where E* arged 

and Ej hoton are the transverse energies deposited by charged hadrons and photons, respec- 
tively, and EJj! eutral is the transverse energy deposited by neutral particles other than photons. 
We identify muons and electrons as isolated when I re i < 0.125 and I re \ < 0.1, respectively. The 
requirement on the relative isolation for electrons is tighter than for muons because the back- 
grounds for electrons are higher than for muons. Electron candidates in the transition region 
between ECAL barrel and endcap (1.44 < \tj\ < 1.57) are excluded because the reconstruction 
of an electron object in this region is not optimal. We require a missing transverse momentum 
Et of at least 40 GeV. The Et is calculated as the absolute value of the vector sum of the px of 
all reconstructed objects. Jets are required to have a px > 30 GeV. The jet energies are corrected 
to establish a uniform response of the calorimeter in rj and a calibrated absolute response in 
px- Furthermore, a correction is applied to take into account the energy clustered in jets due to 
additional proton interactions in the same bunch crossing. 

The observed data are compared to simulated data generated with POWHEG 301 H23ll24"l for the 
single-top process, pythia 6.4.22 [ZQ for the diboson processes, and MadGraph 5.1.1 ||26] 
for the signal and other standard model processes. The POWHEG and MADGRAPH genera- 
tors are interfaced with pythia for the decay of the particles as well as the hadronization and 
the implementation of a CMS custom underlying event tuning (tune Z2) [27|. The matching 
of the matrix-element partons to the parton showers is obtained using the MLM matching 
algorithm J28J. The CTEQ6L1 leading-order (LO) parton distributions are used in the event 
generation [29). The generated events are passed through the CMS detector simulation based 
on Geant4 [30], and then processed by the same reconstruction software as the collision data. 
The simulated events are reweighted to match the observed distribution of the number of si- 
multaneous proton interactions. For the full dataset collected in 2011, we observe on average 
about nine interactions in each event. We smear the jet energies in the simulation to match 
the resolutions measured with data |31|. At least one of the jets within the tracker acceptance 
(\rj | < 2.4) needs to be identified as a b jet. For the b-jet identification, we require the signed 
impact parameter significance of the third track in the jet (sorted by decreasing significance) to 
be larger than a value chosen such that the probability for a light quark jet to be misidentified 
as a b jet is about 1%. We apply scale factors measured from data to the simulated events to 
take into account the different b-jet efficiency and the different probability that a light quark or 
gluon is identified as a b jet in data and simulation 1132 1 . 

The top-quark pair as well as the W and Z production cross section values used in the analysis 
correspond to the measured values from CMS H33l |34|. We use the predicted cross section 
values for the single- top, tt + W, tt + Z, and same-sign WW processes [351438I . The cross section 
values for the diboson production are obtained with the MCFM generator 11391 |40l . 

For the pair-production of the fourth-generation quarks we use the approximate next-to-next- 
to-leading-order cross section values from [41 j. For the electroweak production processes men- 
tioned above, we rescale the next-to-leading-order (NLO) cross sections at 14 TeV ||421| to 7 TeV 
using a scale factor defined as the ratio of the LO cross section at 7 TeV and the LO cross sec- 
tion at 14 TeV as obtained by the MADGRAPH event generator. The resulting production cross 
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sections are maximal, hence assuming | Vjjy 
to the value of A. 

4 Event classification 



Vt'bl = \Vv\>>\ = 1, and are rescaled according 



Different channels are defined according to the number of W bosons in the final state. Given 
that the t' decay mode is the same as the top-quark decay mode, the t'b and t'V processes 
will yield signatures that are very similar to respectively the single-top and tt processes in 
the standard model. We select these processes through the single-lepton decay channel. In 
the signal final states that contain a b' quark, we expect three or four W bosons. If two or 
more of these W bosons decay to leptons, we may have events with two leptons of the same 
charge or with three charged leptons. Although the branching fraction of these decays is small 
compared to that of other decay channels, these final states are very interesting because of the 
low background that is expected from standard model processes. 

4.1 The single-electron and single-muon decay channels 

On top of the aforementioned event selection criteria, we veto events with additional electrons 
or muons with I K \ < 0.2 and pj > 10 GeV for muons and pj > 15 GeV for electrons. We divide 
the selected single-lepton events into different subsamples according to the signal final states. 
Therefore, we define a procedure to count the number of W-boson candidates. Each event has 
at least one W boson that decays to leptons, consistent with the requirements of an isolated 
lepton and a large missing transverse momentum from the neutrino that escapes detection. 
The decays of W bosons to, qq final states are reconstructed with the following procedure. For 
each event, we have a collection of selected jets used as input for the reconstruction of the 
W-boson candidates. The one or two jets that are identified as b jets are removed from the 
collection. W-boson candidates are constructed from all possible pairs of the remaining jets in 
the collection. We use both the expected mass, mfy = 84.3 GeV, and the width, o^L = 9.6 GeV 
from a Gaussian fit to the reconstructed mass distribution of jet pairs from the decay of a W 
boson in simulated tt events. The W-boson candidate with a mass that matches the value 
of best, is chosen as a W boson if its mass is within a ±lc^ window around m%. The 
jet pair that provided the hadronically decaying W boson is removed from the collection and 
the procedure is repeated until no more candidates are found for W bosons decaying to jets. 
Different exclusive subsamples are defined according to the number of b jets (exactly one or at 
least two) and the number of W-boson candidates (one, two, three, and at least four). There 
are seven subsamples, because we do not consider the subsample with only one b jet and one 
W boson. The subsample with two b jets and one W boson is dominated by singly produced 
t' events. In this subsample, we apply a veto for additional jets with a transverse momentum 
exceeding 30 GeV. Furthermore, since bb background tends to have jets that are produced back- 
to-back with balanced pj, we remove this background by requiring A(p(ji,j2) < f + 7t(\pj — 

P?I)/(P? + P?)- 

Table [I] summarizes the requirements that define the different single-lepton decay subsamples, 
after the criteria on the JEj, and the lepton and jet pj and r\ are applied. 

4.2 The same-sign dilepton and trilepton decay channels 

The transverse momentum of at least one of the leptons in the multilepton channel is required 
to be larger than 40 GeV, while the threshold is reduced to 20 GeV for additional leptons. Events 
with two muons or electrons with a mass within 10 GeV of the Z-boson mass are rejected to 
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Table 1: Overview of the event selection requirements defining the different subsamples in 
the single-lepton decay channel. The single-lepton decay channel is divided in seven different 
subsamples according to the number of b jets and the number of W-boson candidates. 
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reduce the standard model background with Z bosons in the final state. We require at least 
four jets for the same-sign dilepton events. In the case of the trilepton events the minimum 
number of required jets is reduced to two. Table |2]summarizes the event selection requirements 
defining the same-sign dilepton and trilepton decay channels that are applied on top of the 
other requirements on the and lepton and jet pj and rj. 

Table 2: Overview of the event selection requirements specific to the same-sign dilepton and 
trilepton decay channels. 
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There are several contributions to the total standard model background for the same-sign dilep- 
ton events. One of these contributions comes from events for which the charge of one of the 
leptons is misreconstructed. Secondly there are events with one prompt lepton and one non- 
prompt lepton passing the isolation and identification criteria. Finally there is an irreducible 
contribution from standard model processes with two prompt leptons of the same sign; e.g. 
W ± W ± / WZ, ZZ, tf + W and tf + Z. Except for W ± W ± / these processes are also the main contri- 
butions to the total background for the trilepton subsample. The event yields for the irreducible 
component of the background for the same-sign dilepton channel and the total background in 
the case of the trilepton subsample are taken from the simulation. We obtain from the data the 
predicted number of background events for the first two contributions to the total background 
in the same-sign dilepton subsample. 

For the same-sign dilepton events with at least one electron, the background is estimated from 
control samples. We determine the charge misidentification rate for electrons using a double- 
isolated-electron trigger. We require two isolated electrons with the dielectron invariant mass 
within 10 GeV of the Z-boson mass. We select events with < 20 GeV and a transverse mass 

Mj = \JlpjEj[\ — cos(Atp(£, #t))] less than 25 GeV to suppress background from top-quark 
and W+jets events. We define the charge misidentification ratio R as the number of events with 
two electrons of the same sign divided by twice the number of events with two electrons of 
opposite sign, i.e. R = Nss/ZNqs- We obtain 0.14% and 1.4% for barrel and endcap electron 
candidates, respectively. After the full event selection is applied, with the exception of the 
electron sign requirement, we obtain a number of selected data events with two electrons and 
with an electron and a muon in the final state. The background with two electrons or with 
an electron and a muon with the same sign is obtained by taking the number of opposite-sign 
events and scaling it with R. 
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5 Setting lower limits on the fourth-generation quark masses 



Another important background contribution to the same-sign dilepton channel originates from 
jets being misidentified as an electron or a muon. We require at least one electron or muon with 
looser isolation (7 re i < 0.2) and identification criteria ("loose lepton"). Additionally, we require 
£t < 20 GeV and Mj < 25 GeV to suppress background from top-quark and W+jets events. 
Moreover, we veto events with leptons of the same flavor that have a dilepton mass within 
20 GeV of the Z boson mass. We count the number of loose and "tight" (regular isolation and 
identification criteria) leptons with a pj below 35 GeV. The threshold on the pj is required to 
suppress contamination from W+jets events. The probability that a loose (L) lepton passes the 
tight (T) selection criteria is then given by the ratio 6tl = Nj/ Nl- To estimate the number of 
events from the background source with a non-prompt lepton, we count the number of events 
in data that pass the event selection criteria with one lepton passing the tight selection criteria 
and a second lepton passing the loose but not the tight criteria. This yield is multiplied by 
£ tl(1 — £tl) to determine the number of events with a non-prompt lepton in the analysis. 

The total number of expected background events for the same-sign dilepton and trilepton chan- 
nels is given in Table [3] 



Table 3: The prediction for the total number of background events compared with the number 
of observed events in the same-sign dilepton and the trilepton subsamples. The number of 
expected signal events are also shown for two possible scenarios. 
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0.96 ±0.12 


Signal (A = 1, m q , = 550 GeV) 


3.31 ±0.15 


2.03 ± 0.36 


5.29 ±0.19 


3.37 ±0.16 


Signal (A = 0.8, m q , = 550 GeV) 


3.79 ±0.15 


2.29 ± 0.36 


6.00 ±0.19 


3.65 ±0.16 



5 Setting lower limits on the fourth-generation quark masses 

We have defined different subsamples according to the reconstructed final state. In each of 
the different subsamples, we reconstruct observables that are sensitive to the presence of the 
fourth-generation quarks. These observables are used as input to a fit of the combined distri- 
butions for the standard model (background-only) hypothesis and the signal-plus-background 
hypothesis. With the profile likelihood ratio as a test statistic, we calculate the 95% confidence 
level (CL) upper limits on the combined input cross section of the signal as a function of the 
^CKM P aram eter A and the mass of the fourth-generation quarks. 

5.1 Observables sensitive to the fourth-generation quark production 

The expected number of events is small in the subsamples with two leptons of the same sign, 
the trilepton subsample, and the two single-lepton subsamples with four W-boson candidates. 
As a consequence, the event counts in each of these subsamples are used as the observable. 
Table [3] summarizes the event counts for the subsamples with two leptons of the same sign and 
the trilepton subsample. 

In the single-lepton subsamples with one or three W bosons, we use Sj as the observable to 
discriminate between the standard model background and the fourth-generation signal, where 
Sj is defined as the scalar sum of the transverse momenta of the reconstructed objects in the 
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final state, namely: 

N wi 

s T = je T + P £ + p»+p ; T + (!) 

where the sum runs over the number of reconstructed hadronically decaying W bosons; p T 
is the pi of the lepton, p\ the pj of the b jet, pj the pj of the second b jet or, if there is no 
additional jet identified as a b jet, the pj of the jet with the highest transverse momentum in the 

W- 

event that is not used in the W-boson reconstruction, and p T n the pj of the i reconstructed 
W boson decaying to jets. In general, the decay products of the fourth-generation quarks are 
expected to have higher transverse momenta compared to the standard model background. 
This is shown in Fig. [T] for three of the subsamples. The dominant contribution to the selected 
signal events in the subsample with two b jets and one W boson would come from the t'b 
process. Almost no signal events are selected for A = 1, because in that case the production 
cross section of t'b is equal to zero. The subsamples with two W bosons are dominated by tt 
events. In this case we use two sensitive observables, St and the mass of the hadronic bW 
system, mbw- The latter observable is sensitive to the fourth generation physics, because of 
the higher mass of a hypothetical fourth generation t' quark compared to the top-quark mass. 
To obtain a higher sensitivity with the m^ w observable, four jets need to be assigned to the 
quarks to reconstruct the final state t'F — > WbWb — > qqbivfb. Therefore, six observables 
with discriminating power between correct and wrong jet/ quark assignments are combined 
with a likelihood ratio method. These observables are angles between the decay products, 
the W-boson mass, the transverse momentum of the top quark decaying to hadrons, and an 
observable related to the values of the b-jet identification variable for the jets. The jet/ quark 
assignment with the largest value of the likelihood ratio is chosen. The mass of the bW system 
is then reconstructed from this chosen jet/quark assignment. The lower plots in Fig. [T] show 
the projections of the two-dimensional Sj versus distribution. 

An overview of the observables used in the fit for the presence of the fourth-generation quarks 
is presented in Table |4j 

Table 4: Overview of the observables used in the limit calculation. 



subsample 



observable 



single-lepton 1W 
single-lepton 2W 
single-lepton 3W 
single-lepton 4W 
same-sign dilepton 
trilepton 



S T and m bw 
S T 

event yield 
event yield 
event yield 



5.2 Fitting for the presence of fourth-generation quarks 

We construct a single histogram "template" that contains the information of the sensitive ob- 
servables from all the subsamples. Different template distributions are made for the signal 
corresponding to the different values of A and the fourth-generation quark masses m^. The 
binning of the two-dimensional observable distribution in the single-lepton subsamples with 
two W bosons is defined using the following procedure. We use a binning in the dimension of 
7«bw such that the top-quark pair background events are uniformly distributed over the bins. 
Secondly, the binning in the dimension of St in each of the bins is chosen to obtain uni- 
formly distributed top-quark pair events also in this dimension. 
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Figure 1: The St distribution for the subsamples with two b jets and one W boson (a), one b 
jet and three W bosons (b), two b jets and two W bosons (c), and the mbw distribution for the 
subsample with two b jets and two W bosons (d). The data distributions of these observables 
are compared to their expectation from the simulation assuming the fitted nuisance parameters. 
The fitted values of the nuisance parameters represent the systematic shifts that are applied 
on the simulation to fit the data in the background-only hypothesis. As an illustration the 
total uncertainty band is shown around the simulated expected distribution before taking into 
account the fitted values of the nuisance parameters. The expected distribution for a signal is 
shown for two different values of the V ( ^J I parameter A and for b' and t' masses of 550 GeV. 
The cross section of the signal in the plots is scaled by a factor of eight for visibility. The last 
bin in all the histograms includes the overflow. We do not expect much signal for A = 1 in (a), 
because the subsample with two b jets and one W boson is mainly sensitive to single t' quark 
production. 



5.3 Results and discussion 
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The templates of the sensitive observables are used as input to obtain the likelihoods for the 
background-only and the signal-plus-background hypotheses. Systematic uncertainties are 
taken into account by introducing nuisance parameters that may affect the shape and the nor- 
malization of the templates. In a case where the systematic uncertainty alters the shape of 
the templates, template morphing |43l |44) is used to interpolate linearly on a bin-by-bin basis 
between the nominal templates and systematically shifted ones. 

The normalization of the templates is affected by the uncertainty in the integrated luminosity 
the lepton efficiency and the normalization of the background processes. The integrated lumi- 
nosity is measured with a precision of 2.2% 11451 and has the same normalization effect on all the 
templates. The uncertainties in the lepton efficiency are a combination of the trigger, selection 
and identification efficiencies that amount to 3% and 5% for muon and electron respectively. 
For the uncertainty in the normalization of the background processes, we use the uncertainties 
in the production cross section of the various standard model processes. The most important 
contributions that affect the normalization of the templates are the 12% |l33l| (30%) uncertainty 
for the top-quark pair (single-top) production cross section and a 50% uncertainty for the W 
production cross section because of the large fraction of selected events with jets from heavy 
flavor quarks. For the multilepton channel, we take into account the uncertainties in the back- 
ground estimation obtained from the data. We also include the uncertainties in the production 
cross sections of Z (5% Ell), WW (35%), WZ (42%), ZZ (27%), tt+W (19%), tt + Z (28%) and 
W ± W ± (49%). The uncertainties in the normalization of diboson and top quark pair production 
in association with a boson are taken from a comparison of the NLO and the LO predictions. 

The largest systematic effects on the shape of the templates originate from the jet energy cor- 
rections [31J and the scale factors between data and simulation for the b-jet efficiency and the 
probability that a light quark or gluon is identified as a b jet [32 j. These effects are estimated 
by varying the nominal value by ±1 standard deviation. The uncertainty in the jet energy res- 
olution of about 10% has a relatively small effect on the expected limits. The same is true for 
the uncertainty in the modeling of multiple interactions in the same beam crossing. The latter 
effect is evaluated by varying the average number of interactions in the simulation by 8%. 

The probability density functions of the background-only and the signal-plus-background hy- 
pothesis are fitted to the data to fix the nuisance parameters in both models. In the signal- 
plus-background model, an additional variable, defined as the cross section for the fourth- 
generation signal obtained by combining the separate search channels, is included. In the com- 
bined cross section variable the relative fraction of each fourth-generation signal process is 
fixed according to the probed model parameters (A, m„>)- Using a Gaussian approximation for 
the probability density function of the test statistic, we determine the 95% CL expected and 
observed limits on the combined cross section variable using the CL S criterion [46-48]. We 
exclude the point (A, m q /) at the 95% CL if the upper limit on the combined cross section vari- 
able is smaller than its predicted value within the fourth-generation model. The procedure is 
repeated for each value of A and m q /. 

5.3 Results and discussion 

We use the CL S procedure to calculate the combined limit for the single muon, single elec- 
tron, same-sign dilepton and trilepton channels. When the value of the V^j^j parameter A 
approaches unity, the standard model single-top and the t'b' processes reach their maximal 
values for the production cross section. When the value of A decreases, the cross section of 
these processes decreases linearly with A. At the same time the expected cross section of the 
t'b and tb' processes increases with (1 — A) and is equal to zero for A = 1. Therefore, the t'b 
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and tb' processes are expected to enhance the sensitivity for fourth-generation quarks when 
the parameter A decreases. This is visible in the upper part of Fig. [2] where both the expected 
and observed limits on m q / are more stringent for smaller values of A. For instance, the limit 
on the fourth-generation quark masses increases by 70 GeV for A = 0.9 compared to the value 
of the limit for A ~ 1. While the t'b and tb' processes do not contribute for A ~ 1, the inclusion 
of the t'b' process results in a more stringent limit (a difference of about 30 GeV) compared to 
when this process is not taken into account. 

The existence of fourth-generation quarks with degenerate masses is excluded for all parame- 
ter values below the line using the assumed model of the matrix. In particular, fourth- 
generation quarks with a degenerate mass below 685 GeV are excluded at the 95% CL for a 
parameter value of A ~ 1. It is worth noting that no limits can be set for A exactly equal to 
unity (A = 1), because in this special case the fourth-generation quarks would be stable in the 
assumed model. The analysis is however valid for values of A extremely close to unity. The 
distance between the primary vertex and the decay vertex of the fourth-generation quarks is 
less than 1 mm for 1 — A > 2 x 10" 14 , a number obtained using the LO formula for the decay 
width of the top quark in which the top-quark mass is replaced with a fourth-generation-quark 
mass of 600 GeV. 

Up to now, the masses of the fourth-generation quarks were assumed to be degenerate. How- 
ever, if a fourth generation of chiral quarks exists, this is not necessarily the case. Therefore, 
it is interesting to study how the limit would change for non-degenerate quark masses. If 
we assume non-degenerate masses, another decay channel for the fourth-generation quarks is 
possible. Namely, the branching fraction for the decay of t' (b') into b' (t') and an off-shell W 
boson becomes non-zero. For values of the mass splitting up to about 25 GeV, this branching 
fraction is small as noted in the introduction. We assume a mass splitting of 25 GeV and un- 
changed branching fractions for the t' and b' decays. The sensitivity of the analysis increases 
or decreases depending on the specific values of the masses and hence the production cross 
sections of the fourth-generation quarks. The effect of the mass difference between the fourth- 
generation quarks on the exclusion limit is shown in the bottom plot of Fig. 2] for a V^j^ pa- 
rameter A ~ 1. For instance in case m t i = my + 25 GeV (m t i = my — 25 GeV), the limit on m t i 
increases about +20 (—20) GeV with respect to the degenerate-mass case. To obtain this limit, 
we do not take into account the electroweak t'b' process, which results in more conservative 
exclusion limits. In particular one observes that quarks with degenerate masses below about 
655 GeV are excluded at the 95% CL compared to 685 GeV when the t'b' process is included. 



6 Summary 

Results from a search for a fourth generation of quarks have been presented. A simple model 
for a unitary CKM matrix has been defined based on a single parameter A = \Vfo\ 2 = \V t iy\ 2 . 
Degenerate masses have been assumed for the fourth-generation quarks, hence m t i = my. 
The information is combined from different subsamples corresponding to different final states 
with at least one electron or muon. Observables have been constructed in each of the subsam- 
ples and used to differentiate between the standard model background and the processes with 
fourth-generation quarks. With this strategy the search for singly and pair-produced t' and b' 
quarks has been combined in a coherent way into a single analysis. Model-dependent limits 
are derived on the mass of the quarks and the matrix element A. The existence of fourth- 
generation quarks with masses below 685 GeV is excluded at 95% confidence level for minimal 
off-diagonal mixing between the third- and the fourth-generation quarks. A non-zero cross 
section for the single fourth-generation quark production processes, corresponding to a value 
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Figure 2: Top: Exclusion limit on m t r = my as a function of the parameter A. The 

parameter values below the solid line are excluded at 95% CL. The inner (outer) band indicates 
the 68% (95%) confidence interval around the expected limit. The slope indicates the sensitivity 
of the analysis to the t'b and tb' processes. Bottom: For a parameter value A <~ 1, the 

exclusion limit on m t / versus m t i — my is shown. The exclusion limit is calculated for mass 
differences up to 25 GeV. The existence of up-type fourth-generation quarks with mass values 
below the observed limit are excluded at the 95% CL. 



12 



References 



of the Vq^jJi parameter A < 1 gives rise to a more stringent limit. When a mass difference of 
25 GeV is assumed between t' and b' quarks, the limit on m t ' shifts by about +20 (—20) GeV for 
mf = my + 25 GeV (m t / = my — 25 GeV). These results significantly reduce the allowed param- 
eter space for a fourth generation of fermions and raise the lower limits on the masses of the 
fourth generation quarks to the region where nonperturbative effects of the weak interactions 
are important. 
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